Systems and methods for identifying an event in data

ABSTRACT

The present disclosure includes systems, apparatuses, and methods for event identification. In some aspects, a method includes receiving data including text and performing natural language processing on the received data to generate processed data that indicates one or more sentences. The method also includes generating, based on a first keyword set, a second keyword set having more keywords than the first keyword set. The method further includes, for each of the first and second keyword sets: detecting one or more keywords and one or more entities included in the processed data, determining one or more matched pairs based on the detected keywords and entities, and extracting a sentence, such as a single sentence or multiple sentences, from a document based on the one or more sentences indicated by the processed data. The method may also include outputting at least one extracted sentence.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to U.S. Provisional ApplicationNo. 62/773,502, filed Nov. 30, 2018, and entitled, “CEREAL: AConsolidated System for Robust Multi-Document Entity Risk Extraction andTaxonomy Augmentation,” the disclosure of which is incorporated byreference herein in its entirety.

TECHNICAL FIELD

The present subject matter is directed generally to eventidentification, and more particularly but without limitation, toidentifying or predicting a risk event in textual data.

BACKGROUND

Identifying or predicting risk events in textual data associated withindividuals, companies, and other entities is a common natural languageprocessing (NLP) task known as risk mining. Monitoring systems rely onrisk mining to describe risk events that are passed on to an expert foranalysis.

Risk mining research is often focused on classification of risk fromnon-risk and the features that contribute to the distinction. Forexample, prior risk mining research has considered features such astextual association with stock price movement and financial documents,sentiment features in identifying banking risks, and textual featuresfor risk in news. Additionally, heuristic classifications have been usedin which seed patterns that describe risk relationships are used toaugment a risk taxonomy used in an alert monitoring system for earningsreports. The risk taxonomy can be a source of information in risk miningwhether the result of manual definition, machine learningclassification, or both, crowdsourcing, and/or paraphrase detection.Although risk mining research has been explored, the quality of riskmining results are often less than acceptable and/or require manualreview of the risk mining results to obtain filtered results.Additionally, large quantities of results can make the manual reviewprocess time consuming and delay or prolong timely review of the riskmining results.

SUMMARY

The present application relates to systems and methods for eventidentification and/or prediction of events, such as risk eventidentification and/or prediction in textual data. To illustrate, thesystems and methods may use a predefined keyword taxonomy, entities anddata sources to generate and return text extractions based onbidirectional distances (e.g., token distances) between entities andkeywords. For example, data including text from a data source, such as astreaming data source, news data, a database, or a combination thereof,may be received and natural language processing (NLP) is performed onthe data. In some implementations, the NLP may generate processed datathat indicates one or more sentences. Additionally, or alternatively,the NLP may include tokenization, lemmatization, and/or sentencizationon the data. In a particular implementation, the NPL includes a naturallanguage processing pipeline including (in sequence) a tokenizer, apart-of-speech tagger, a dependency parser, and a named entityrecognizer. It is noted that a dependency-based sentencizer may be usedas compared to a simpler stop-character based approach due to theunpredictable formatting of certain domains of text—e.g. web-mined newsand regulatory filings. After the NLP, a set of keywords are compared tothe processed data to detect keywords included in the processed data.For each detected keyword, a corresponding entity is identified that ispositioned closest to the corresponding keyword to determine a matchedpair for the keyword. In some implementations, a distance between thekeyword and the entity of the matched pair is compared to a thresholddistance for purposes of discarding matched pairs. Based on a particularmatched pair, the systems and methods may extract a sentence, such as asingle sentence or multiple sentences, from a document based thatincludes the matched pair (e.g., the entity and the keyword). Thesystems and methods may also output at least one extracted sentence. Theextracted sentence may be stored or provided to an electronic device forreview and/or analysis.

In some implementations, the systems and methods may also expand aninitial seed taxonomy, such as a keyword set, using word vectorencodings. For example, for at least one document of one or moredocuments corresponding to the data, a corresponding semantic vector maybe generated—e.g., based on a skipgram model that utilizes words andsubwords from the document. For at least one keyword, the at least onekeyword is compared to each of one or more semantic vectors to determinea corresponding similarity score. A semantic vector having a highestsimilarity score to the keyword is identified to determine a term of theidentified semantic vector as a candidate term. In some implementations,the similarity score of the determined semantic vector having a highestsimilarity score is compared to a threshold to determine whether or notto discard the candidate term—e.g., the term is discarded if the scoreis less than or equal to the threshold. The candidate term may be addedto the keyword set to generate the expanded keyword set. The initialkeyword set and/or the expanded keyword may be applied to the processeddata to identify matched pairs and to extract one or more sentences asdescribed above.

Thus, the present disclosure provides the ability to extract events(e.g., risks) at scale, improve the quality of the extractions overtime, grow the keyword taxonomy to increase coverage, and do so withminimum manual effort. The systems and methods described herein alsoinclude a scalable system for finding entity-event pairings which, ascompared to other systems, better incorporate human insights in theinitial system configuration to obviate the need for a riskclassification engine. For example, the systems and methods describedherein provide a hybrid human-automated system that provides scalableevent-monitoring capabilities. Further, without relying on multiplecomplex NLP sub-systems to make connections between entities and eventkeywords, the system may derive maximum benefit from surfaceinformation. Because of the specificity of the initial taxonomy,extracts are assumed to express some degree of entity-risk relationshipsand are based on “shallow” surface parsing rather than deepermorpho-syntactic parsing. Additionally, the degree to which these twodeviations hold is the subject of the evaluation—e.g., comparing to adistance threshold between keyword and entity. To illustrate, thesystems and methods described herein may utilize techniques, such asfiltering and/or thresholding, to address high recall associated withthe predefinition of the seed taxonomy. Additionally, or alternatively,the systems and methods described herein may address high system recallrelative to maintaining flexibility for analyst users and dynamicdefinition of the risk problem space—this may include summarization ofresults for better presentation, alternative source data at thedirection of the analyst for given risk categories, and token distancethresholding.

It is noted that while applied here to the risk mining space, in otherimplementations, the system could be used for any data, entities andtaxonomies to support generalized event monitoring. To illustrate, thesystems and methods may be equally applicable to other areas ofidentification and/or predication, such as document review, auditing,and the like, as illustrative, non-limiting examples.

In one particular aspect, a method for identifying an event in dataincludes receiving data at a receiver from a data source, the dataincluding one or more documents each including text. The method includesperforming natural language processing on the received data to generateprocessed data. The processed data indicates one or more sentences. Themethod includes generating, based on the data and a first keyword set, asecond keyword set having a greater number of keywords than the firstkeyword set. The method also includes, for each keyword set of the firstkeyword set and each keyword in the second keyword set: detecting one ormore keywords and one or more entities included in the processed databased on the keyword set and an entity set, determining one or morematched pairs based on the detected one or more keywords and thedetected one or more entities, extracting a sentence from a documentbased on the one or more sentences indicated by the processed data, andoutputting the extracted sentence. The sentence corresponds to at leastone matched pair of the one or more matched pairs and includes a singlesentence or multiple sentences.

In another aspect, a system may be provided. The system includes a dataingestor configured to receive data at a receiver from a data source andto perform natural language processing on the received data to generateprocessed data. The data includes one or more documents each includingtext, and the processed data indicates one or more sentences. The systemincludes a taxonomy expander configured to generate, based on the dataand a first keyword set, a second keyword set having a greater number ofkeywords than the first keyword set. The system also includes a termdetector configured to detect, for each keyword set of the first keywordset and each keyword in the second keyword set, one or more keywords andone or more entities included in the processed data based on the keywordset and an entity set. The system further includes an output generatorconfigured to, for each keyword set of the first keyword set and thesecond keyword set: determine one or more matched pairs based on thedetected one or more keywords and the detected one or more entities,extract a sentence from a document based on the one or more sentencesindicated by the processed data, and output the extracted sentence. Thesentence corresponds to at least one matched pair of the one or morematched pairs and includes a single sentence or multiple sentences.

In yet another aspect, a computer-based tool may be provided. Thecomputer-based tool may include non-transitory computer readable mediahaving stored thereon computer code which, when executed by a processor,causes a computing device to perform operations that include receivingdata at a receiver from a data source. The data includes one or moredocuments each including text. The operations include performing naturallanguage processing on the received data to generate processed data. Theprocessed data indicates one or more sentences. The operations alsoinclude generating, based on the data and a first keyword set, a secondkeyword set having a greater number of keywords than the first keywordset. The operations further include, for each keyword set of the firstkeyword set and each keyword in the second keyword set: detecting one ormore keywords and one or more entities included in the processed databased on the keyword set and an entity set, determining one or morematched pairs based on the detected one or more keywords and thedetected one or more entities, extracting a sentence from a documentbased on the one or more sentences indicated by the processed data, andoutputting the extracted sentence. The sentence corresponds to at leastone matched pair of the one or more matched pairs and includes a singlesentence or multiple sentences.

The foregoing broadly outlines the features and technical advantages ofthe present invention in order that the detailed description of theinvention that follows may be better understood. Additional features andadvantages of the invention will be described hereinafter which form thesubject of the claims of the invention. It should be appreciated bythose skilled in the art that the conception and specific embodimentdisclosed may be readily utilized as a basis for modifying or designingother structures for carrying out the same purposes of the presentinvention. It should also be realized by those skilled in the art thatsuch equivalent constructions do not depart from the spirit and scope ofthe invention as set forth in the appended claims. The novel featureswhich are believed to be characteristic of the invention, both as to itsorganization and method of operation, together with further objects andadvantages will be better understood from the following description whenconsidered in connection with the accompanying figures. It is to beexpressly understood, however, that each of the figures is provided forthe purpose of illustration and description only and is not intended asa definition of the limits of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawings, in which:

FIG. 1 shows a system configured to perform operations in accordancewith aspects of the present disclosure;

FIG. 2 shows a flow diagram illustrating functionality of the system ofFIG. 1 implemented in accordance with aspects of the present disclosure

FIG. 3 is a block diagram of a system for identifying an event in datain accordance with the present disclosure;

FIG. 4 illustrates a graph of expert preference ratings; and

FIG. 5 is a flow chart illustrating an example of a method ofidentifying an event in data.

DETAILED DESCRIPTION

Various features and advantageous details are explained more fully withreference to the non-limiting embodiments that are illustrated in theaccompanying drawings and detailed in the following description.Descriptions of well-known starting materials, processing techniques,components, and equipment are omitted so as not to unnecessarily obscurethe invention in detail. It should be understood, however, that thedetailed description and the specific examples, while indicatingembodiments of the invention, are given by way of illustration only, andnot by way of limitation. Various substitutions, modifications,additions, and/or rearrangements within the spirit and/or scope of theunderlying inventive concept will become apparent to those skilled inthe art from this disclosure.

FIG. 1 is a block diagram of an exemplary system 100 configured withcapabilities and functionality for event identification. As shown inFIG. 1, system 100 includes server 110, at least one user terminal 160,at least one data source 170, and network 180. These components, andtheir individual components, may cooperatively operate to providefunctionality in accordance with the discussion herein. For example, inoperation according to one or more implementations, data (e.g., textualdata or documents) may be obtained from data sources 170 and may beprovided as input to server 110. The various components of server 110may cooperatively operate to perform event identification and/orprediction. For example, the various components of server 110 maycooperative operate to perform natural language processing (NLP) isperformed on the data In some implementations, the NLP may generateprocessed data that indicates one or more sentences. After the NLP, thevarious components of server 110 may compare a set of keywords to theprocessed data to detect keywords included in the processed data. Foreach detected keyword, a corresponding entity is identified that ispositioned closest to the corresponding keyword to determine a matchedpair for the keyword. In some implementations, a distance between thekeyword and the entity of the matched pair is compared to a thresholddistance for purposes of discarding matched pairs. Based on a particularmatched pair, the various components of server 100 may extract asentence, such as a single sentence or multiple sentences, from adocument based that includes the matched pair (e.g., the entity and thekeyword). The sentence may be stored provided to an electronic devicefor review and/or analysis. As such, various aspects of the presentdisclosure allow event identification (and generation of a textextraction) based on use of a predefined keyword taxonomy, entities anddata sources and based on bidirectional distances (e.g., tokendistances) between entities and keywords detected within the data.

It is noted that the functional blocks, and components thereof, ofsystem 100 of embodiments of the present invention may be implementedusing processors, electronics devices, hardware devices, electronicscomponents, logical circuits, memories, software codes, firmware codes,etc., or any combination thereof. For example, one or more functionalblocks, or some portion thereof, may be implemented as discrete gate ortransistor logic, discrete hardware components, or combinations thereofconfigured to provide logic for performing the functions describedherein. Additionally or alternatively, when implemented in software, oneor more of the functional blocks, or some portion thereof, may comprisecode segments operable upon a processor to provide logic for preformingthe functions described herein.

It is also noted that various components of system 100 are illustratedas single and separate components. However, it will be appreciated thateach of the various illustrated components may be implemented as asingle component (e.g., a single application, server module, etc.), maybe functional components of a single component, or the functionality ofthese various components may be distributed over multipledevices/components. In such aspects, the functionality of eachrespective component may be aggregated from the functionality ofmultiple modules residing in a single, or in multiple devices.

In some aspects, server 110, user terminal 160, and data sources 170 maybe communicatively coupled via network 180. Network 180 may include awired network, a wireless communication network, a cellular network, acable transmission system, a Local Area Network (LAN), a Wireless LAN(WLAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN),the Internet, the Public Switched Telephone Network (PSTN), etc., thatmay be configured to facilitate communications between user terminal 160and server 110.

User terminal 160 may be implemented as a mobile device, a smartphone, atablet computing device, a personal computing device, a laptop computingdevice, a desktop computing device, a computer system of a vehicle, apersonal digital assistant (PDA), a smart watch, another type of wiredand/or wireless computing device, or any part thereof. User terminal 160may be configured to provide a graphical user interface (GUI) via whicha user may be provided with information related to data and informationreceived from server 100. For example, User terminal 160 may receiveresults of event identification and/or prediction from server 100. Theresults may include a match pair including a keyword, an entity, orboth, one or more extracted sentences, a document identifier, or acombination thereof, as illustrative, non-limiting examples. A user mayreview the results and provide an analysis or feedback regarding theresults. The analysis or feedback may be provided to server 100 fromuser terminal 160 as an input.

Data sources 170 may comprise at least one source of textual data. Forexample, the data source(s) may include a streaming data source, newsdata, a database, a social media feed, a data room, another data source,the like, or a combination thereof. In a particular implementation, thedata from data source 170 may include or correspond to one or moreentities. The one or more entities may include an individual, a company,a government, an agency, an organization, the like, or a combinationthereof, as illustrative, non-limiting examples.

Server 110 may be configured to receive data from data source(s) 170, toapply customized natural language processing algorithms and/or otherprocessing to identify one or more events based on the received data.This functionality of server 110 may be provided by the cooperativeoperation of various components of server 110, as will be described inmore detail below. Although FIG. 1 shows a single server 110, it will beappreciated that server 110 and its individual functional blocks may beimplemented as a single device or may be distributed over multipledevices having their own processing resources, whose aggregatefunctionality may be configured to perform operations in accordance withthe present disclosure. In some implementations, server 110 may beimplemented, wholly or in part, on an on-site system, or on acloud-based system.

As shown in FIG. 1, server 110 includes processor 111, memory 112,database 113, data ingestor 120, output generator 122, and, optionally,taxonomy expander 124. It is noted that the various components of server110 are illustrated as single and separate components in FIG. 1.However, it will be appreciated that each of the various components ofserver 110 may be a single component (e.g., a single application, servermodule, etc.), may be functional components of a same component, or thefunctionality may be distributed over multiple devices/components. Insuch aspects, the functionality of each respective component may beaggregated from the functionality of multiple modules residing in asingle, or in multiple devices.

In some aspects, processor 111 may comprise a processor, amicroprocessor, a controller, a microcontroller, a plurality ofmicroprocessors, an application-specific integrated circuit (ASIC), anapplication-specific standard product (ASSP), or any combinationthereof, and may be configured to execute instructions to performoperations in accordance with the disclosure herein. In some aspects,implementations of processor 111 may comprise code segments (e.g.,software, firmware, and/or hardware logic) executable in hardware, suchas a processor, to perform the tasks and functions described herein. Inyet other aspects, processor 111 may be implemented as a combination ofhardware and software. Processor 111 may be communicatively coupled tomemory 112.

Memory 112 may comprise read only memory (ROM) devices, random accessmemory (RAM) devices, one or more hard disk drives (HDDs), flash memorydevices, solid state drives (SSDs), other devices configured to storedata in a persistent or non-persistent state, network memory, cloudmemory, local memory, or a combination of different memory devices.Memory 112 may store instructions that, when executed by processor 111,cause processor 111 to perform operations in accordance with the presentdisclosure. In aspects, memory 112 may also be configured to facilitatestorage operations. For example, memory 112 may comprise database 113for storing one or more keywords (e.g., one or more keyword sets), oneor more entities (e.g., an entity set), one or more thresholds, one ormore matched pairs, one or more semantic vectors, one or more candidateterms, one or more similarity scores, one or more extracted sentence,input (e.g., from user terminal 160), other information, etc., whichsystem 100 may use to provide the features discussed herein. Database113 may be integrated into memory 112, or may be provided as a separatemodule. In some aspects, database 113 may be a single database, or maybe a distributed database implemented over a plurality of databasemodules. In some embodiments, database 113 may be provided as a moduleexternal to server 110. Additionally, or alternatively, server 110 mayinclude an interface configured to enable communication with data source170, user terminal 160 (e.g., an electronic device), or a combinationthereof.

Data ingestor 120 may be configured to receive data at a receiver from adata source, the data comprising one or more documents each comprisingtext, and to perform natural language processing on the received data togenerate processed data, the processed data indicating one or moresentences.

Term detector 121 may be configured to detect, for each keyword set ofthe first keyword set and each keyword in the second keyword set, one ormore keywords and one or more entities included in the processed databased on the keyword set and an entity set.

Output generator 122 may be configured to, for each keyword set of thefirst keyword set and the second keyword set, determine one or morematched pairs based on the detected one or more keywords and thedetected one or more entities, and extract a sentence from a documentbased on the one or more sentences indicated by the processed data,where the sentence corresponds to at least one matched pair of the oneor more matched pairs and comprises a single sentence or multiplesentences. To determine the one or more matched pairs, output generator122 is further configured to for each keyword of the detected one ormore keywords, identify a corresponding entity of the detected one ormore entities that is positioned closest to the corresponding keyword todetermine a matched pair for the keyword, Additionally, output generator122 may be further configured to store and/or output the extractedsentence.

In some implementations, output generator 122 includes a filter 123.Filter 123 is configured to for each of the one or more matched pairsbased on the first keyword set and each of the one or more matched pairsbased on the second keyword set, determine a distance between thekeyword and the entity of the matched pair, perform a comparison betweenthe determined distance and a threshold, and determine to retain thematched pair or discard the matched pair based on whether or not thecomparison indicate the determined distance is greater than or equal tothe threshold. For example, filter 123 may discard the matched pairbased on the determined distance being greater than or equal to thethreshold. Alternatively, filter may retain the matched pair for furtherprocessing and/or consideration based on the distance being less thanthe threshold.

Taxonomy expander 124 may be configured to generate, based on the dataand a first keyword set, a second keyword set having a greater number ofkeywords than the first keyword set. Additional functionality oftaxonomy expander 124 is described further herein at least withreference to blocks 240-248 of FIG. 2. It is noted that thefunctionality of taxonomy expander 124 to expand a keyword set togenerate an expanded keyword set may be used prior to, during, or afterevent identification or prediction.

The database 113 may be coupled to data ingestor 120, term detector 121,output generator 122, taxonomy expander 124, or a combination thereof.In some implementations, database 113 is configured to store the firstkeyword set, the second keyword set, the entity set, the processed data,one or more thresholds, one or more extracted sentences, a plurality ofmatched pairs, or a combination thereof.

The functionality of server 110 will now be discussed with respect tothe block flow diagram illustrated in FIG. 2. FIG. 2 shows a flowdiagram illustrating functionality of system 100 for detecting an eventin data. Blocks of method 200 illustrated in FIG. 2 may be performed byone or more components of system 100 of FIG. 1. For example, blocks 210,212, and 214 may be performed by data ingestor 120, block 216 may beperformed by term detector 121, blocks 218-222 may be performed byoutput generator 122, and blocks 240-248 may be performed by taxonomyexpander 124.

At block 210, data is received (e.g., at a receiver). For example, thedata may include one or more documents and may be received from datasources 170. In some implementations, data sources 170 may include astreaming data source, news data, a database, or a combination thereof.At block 212, a keyword set and an entity set are received. In someimplementations, the keyword set (e.g., seed values) and the entity setmay be received based on user input or retrieved from a memory. At block214, the data is provided to a NLP pipeline to generate processed datathat indicates one or more sentences. In some implementations, the NLPpipeline includes (in sequence) a tokenizer, a part-of-speech tagger, adependency parser, and a named entity recognizer. The dependency parsermay be preferable to other types of sentence detectors, such as astop-character parser, due to the unpredictable formatting of certaindomains of text, such as web-mined news and/or regulatory filings.

At block 216, keyword and entity identification is performed. Forexample, based on a taxonomy, keywords (from the keyword set) may beidentified in a list of tokens. To illustrate, the set of keywords arecompared to the processed data to detect keywords in the processed data.Similarly, entities (from the entity set) may be identified in a list oftokens. At block 218, keyword and entity matching is performed. Forexample, for each detected keyword, a corresponding entity is identifiedthat is positioned closest to the corresponding keyword to determine amatched pair for the keyword. The closest entity may be before or afterthe keyword, and may be in the same sentence or a different sentence.

At block 220, matched pair filtering is performed. For example, adistance (in tokens) between the keyword and the entity of a matchedpair is determined, and if the distance is greater than or equal to athreshold, the matched pair is discarded (e.g., filtered out). At block222, a sentence output result is generated. For example, a sentence thatcontains a matched pair may be extracted and output. The sentence may bea single sentence (if the keyword and the entity are in the samesentence) or multiple sentences (if the keyword and the entity are indifferent sentences). The extracted sentence may be output to anelectronic device for display to a user for review and/or analysis orthe extracted sentence may be stored in a memory for later processing.

Method 200 also enables expansion of an initial seed taxonomy. Toillustrate, at block 240, semantic vectors are generated. For example,for at least one document of the received data, a corresponding semanticvector may be generated. In some implementations, the semantic vectormay be generated based on a skipgram model that utilizes words andsubwords from the document. At block 242, a similarity calculation isperformed. For example, at least one keyword is compared to each of thegenerated semantic vectors to determine corresponding similarity scores.

At block 244, candidate term identification is performed. For example, asemantic vector having a highest similarity score to the keyword isidentified to identify a term of the semantic vector as a candidateterm. Optionally, at block 246, candidate terms are filtered. Forexample, the similarity score of the candidate term is compared to athreshold to determine whether or not to discard the candidate term(e.g., the candidate term is discarded if the score is less than orequal to the threshold). At block 248, the taxonomy is expanded. Forexample, one or more candidate terms are added to the taxonomy togenerate the expanded taxonomy (e.g., an expanded keyword set). Theexpanded taxonomy may be used in performing keyword-entity matching, asdescribed with reference to the operations of blocks 212-222.

In some implementations of FIG. 1, system 100 provides ahybrid-automated system that leverages human insight at the outset withexpert (a) defined entities; (b) defined data sources; and (c) definedkeyword taxonomy. For example, by leveraging (a-c), system 100 (e.g.,server 110) may return extractions based on bidirectional entity keywordsurface distances for expert evaluation. For example in examplesentences (1) below, the risk term “pipe bomb” is a risk associated withthe entity Time Warner.

-   -   (1)(a) Later Wednesday, CNN received a pipe bomb at its Time        Warner Center headquarters at Manhattan sent to ex-CIA director        John Brennan and a suspicious package sent to Rep. Maxine Waters        . . . .    -   (1)(b) On Monday, a pipe bomb was found in a mailbox at the        Katonah home of billionaire business magnet and political        activist George Soros.

Referring to sentences (1)(a) and (1)(b), system 100 may return both thesingle sentence (1)(a) and multi-sentences (1)(a-b) without the need fora separate risk classification engine or more complex NLP. System 100may also increase taxonomy coverage by encoding word vectors trained onthe same sources of data (suspicious package in (1)(a)). Accordingly,server 110 may return high value information to analysts collaboratingwith server 100 It is noted that the operations and functionalityprovided by server 110 combines the efficiencies that tuned MachineLearning (ML) systems can offer with the rich depth of experience andinsight that analysts bring to the process.

Thus, system 100 (e.g., server 110) and its corresponding operations andfunctions, provides the ability to extract events (e.g., risks) atscale, improve the quality of the extractions over time, grow thekeyword taxonomy to increase coverage, and do so with minimum manualeffort. System 100 also includes a scalable system for findingentity-event pairings which, as compared to other systems, betterincorporate human insights in the initial system configuration toobviate the need for a risk classification engine. For example, system100 provides a hybrid human-automated system that provides scalableevent-monitoring capabilities. Further, without relying on multiplecomplex NLP sub-systems to make connections between entities and eventkeywords, system 100 (e.g., server 110) may derive maximum benefit fromsurface information. Because of the specificity of the initial taxonomy,extracts are assumed to express some degree of entity-risk relationshipsand are based on “shallow” surface parsing rather than deepermorpho-syntactic parsing. Additionally, the degree to which these twodeviations hold is the subject of the evaluation—e.g., comparing to adistance threshold between keyword and entity. To illustrate, system 100may utilize the techniques described herein, such as filtering and/orthresholding, to address high recall associated with the predefinitionof the seed taxonomy. Additionally, or alternatively, system 100 mayaddress high system recall relative to maintaining flexibility foranalyst users and dynamic definition of the risk problem space—this mayinclude summarization of results for better presentation, alternativesource data at the direction of the analyst for given risk categories,and token distance thresholding.

A system of the present disclosure is a custom NLP processing pipelinecapable of the ingesting and analyzing hundreds of thousands of textdocuments. The system includes at least four components:

-   -   1. Document Ingest and Processing: Raw text documents are read        and tokenization, lemmatization, and sentencization are        performed.    -   2. Keyword/Entity Detection: Instances of both keywords and        entities are identified in the processed text, and each risk        keyword occurrence is matched to the nearest entity token.    -   3. Match Filtering and Sentence Retrieval: Matches within the        documents are filtered and categorized by pair distance and/or        sentence co-occurrence, and the filtered sentences are retrieved        for context.    -   4. Semantic Encoding and Taxonomy Expansion: A semantic        vectorization algorithm is trained on domain-specific text and        used to perform automated expansion of the keyword taxonomy.

This design architecture allows for significant customization, highthroughput, and modularity for uses in experimental evaluation anddeployment in production use-cases. The system may support decentralizedor streaming architectures, with each document being processedindependently and learning systems (specifically at the semanticencoding/expansion steps) configured for continuous learning or batchmodel training.

One or more known systems can be used for document ingest and low levelNLP, such as spaCy, as a non-limiting example. For example, a defaultNLP pipeline may run tokenizer→part-of-speech tagger→dependencyparser→named entity recognizer.

Sentence breaks found by the dependency parser may be used to annotateeach of the found keyword-entity pairs as being either in the same ordifferent sentences. A dependency-based sentencizer is preferred to asimpler stop-character based approach due to the unpredictableformatting of certain domains of text, e.g., web-mined news andregulatory filings.

The system may allow for a text generator object to be provided, and maytake advantage of multi-core processing to parallelize batching. In suchan implementation, each processed document piped in by the system isconverted to its lemmatized form with sentence breaks noted so thatsentence and multi-sentence identification of keyword/entity distancescan be captured.

In the absence of intervening information or a more sophisticatedapproach to parsing, the mention of an entity and risk keyword in aphrase or sentence is the most coherent semantically and pragmatically(as well as morpho-syntactically). For example, example sentence (2)below describes the entity Verizon and its litigation risk associatedwith lawsuit settlement (keywords being settle and lawsuit).

-   -   (2) In 2011, Verizon agreed to pay $20 million to settle a        class-action lawsuit by the federal Equal Employment Opportunity        Commission alleging that the company violated the Americans with        Disabilities Act by denying reasonable accommodations for        hundreds of employees with disabilities.

Returning the entire sentence yields additional information—the lawsuitis class-action and the allegation in the complaint is that Verizon“denied reasonable accommodations for hundreds of employees withdisabilities.” The detection process performed by the system begins bytesting for matches of each keyword with each entity, for every possiblekeyword-entity pairing in the document. Algorithm 1 provides thesimplified pseudocode for this process.

Algorithm 1 Entity-Keyword Pairing Require: taxonomy and entities listsfor keyword in taxonomy do  for entity in entities do   keywordLocs=-findLocs(keyword)   entityLocs = findLocs(entity)   for kLoc inkeywordLocs do    bestHit = findClosestPair(kLoc, entityLocs)   results.append((keyword, entity, bestHit))   end for  end for end forreturn findClosestPair is two token indicies

In some implementations, for every instance of every keyword, thenearest instance of every available entity is paired—regardless ofwhether it precedes or proceeds the keyword. Furthermore, an entity maybe found to have multiple risk terms associated with it, but eachinstance of a risk term will only apply itself to the closest entity.This helps prevent overreaching conclusions of risk while allowing thesystem to remain flexible. For example, example sentence (3) extends theextract of example sentence (2) to the prior contiguous sentence whichcontains settlement. This extension provides greater context forVerizon's lawsuit. Example sentence (3) is actually background for alarger proposition being made in the document that Verizon is inviolation of settlement terms from a previous lawsuit.

-   -   (3) McDonald says this treatment violated the terms of a        settlement the company reached a few years earlier regarding its        treatment of employees with disabilities. In 2011, Verizon        agreed to pay $20 million to settle a class-action lawsuit by        the federal Equal Employment Opportunity Commission . . . .

The “shallow” parsing approach of the system promotes efficiency and ispreferable to more complex NLP, e.g., chunking or co-referenceresolution. Nonetheless, this flexibility comes at a computational cost:a total of (m·a)×(n·b) comparisons must be made for each document, wherem is the number of keyword terms across all taxonomic categories, a theaverage number of instances of each keyword per document, n the numberof entities provided, and b the average number of entity instances perdocument. Changing any single one of these variables will result incomputational load changing with O(n) complexity, but their cumulativeeffects can quickly add up. For parallelization purposes, each keywordis independent of each other keyword and each entity is independent ofeach other entity. This means that in an infinitely parallel(theoretical) computational scheme, the system runs on O(a×b), whichwill vary as a function of the risk and text domains.

After keyword-entity pairing, the system has completed a substantialpart of document processing and risk identification. The next componentseeks to (a) filter away results unlikely to hold analytic value; and(b) identify the hits as being either single sentence or multi-sentenceusing the sentence information. The first of these goals may be achievedwith a simple hit distance cutoff where any keyword-entity pair withmore than a particular count of intervening tokens is discarded. Aparticular setting of a hard cutoff (e.g., a particular amount) improveskeyword-entity spans by not including cross-document matches for largedocuments. Once filtering is complete, the system uses document sentencebreaks to determine the membership of each keyword and entity for eachpairing and, ultimately, whether they belong to the same sentence.

The system may automate term expansion by using similarity calculationsof semantic vectors. These vectors are generated by training a skipgrammodel, which relies on words and subwords from the same data source asthe initial extractions. This ensures that domain usage of language iswell-represented, and any rich domain-specific text may be used to trainsemantic vectors.

For each taxonomic risk term encountered, the model vocabulary for theminimized normalized dot product

$\frac{r \cdot w}{{{r}{w}}}$

(e.g., a basic similarity score) is searched, and the system returns thetop-scoring vocabulary terms as candidates for taxonomic expansion.

EXAMPLE

FIG. 3 shows an example system 300 in accordance with the presentdisclosure. The system 300 includes a document ingest 302, a semanticexpansion 304, a keyword taxonomy 306, an expanded keyword taxonomy 308,a tokenizer and sentencizer 310, an entity detector 312, an entity list314, a keyword location detector 316, keyword sentences 318, akeyword-based selector 320, a keyword entity pairing 322, a filter 324,a same sentence match pool 326, a cross-sentence match pool 328, acomparator 330, and method rankings 332.

Document ingest 302 includes a corpus of documents (e.g., one or moredocuments) that are ingested into system 300 (e.g., via a documentingestor). The document corpus is provided as a semantic model trainingcorpus to semantic expansion 304. Semantic expansion 304 is adomain-specific semantic expander configured to expand a taxonomy basedon the semantic model training corpus. For example, semantic expansion304 may receive keyword taxonomy 306 as an input of seed terms. Keywordtaxonomy 306 may be generated by one or more users who enter keywordsand related categories. Semantic expansion 304 may expand the seed terms(e.g., the keyword taxonomy 306) based on the input documents togenerate the expanded keyword taxonomy 308. For example, semanticexpansion 304 may generate one or more semantic vectors and identifysemantic vectors (corresponding to new keywords) for inclusion in secondkeyword taxonomy 308 based on similarity scores of the semantic vectors.The expanded keyword taxonomy 308 may include more detailed keywords invarious categories, as well as common misspellings of keywords, suchthat words intended as keywords may be identified by system 300. Theexpanded keyword taxonomy 308 may be combined with the keyword taxonomy306 to generate a taxonomy used by system 300. Additionally, oralternatively, the expanded keyword taxonomy 308 and the keywordtaxonomy 306 may be separately used to provide for an ability to rateeach taxonomy individually.

In addition to being provided to semantic expansion 304, the ingesteddocuments (e.g., from document ingest 302) are provided to tokenizer andsentencizer 310. Tokenizer and sentencizer 310 is configured to detectone or more tokens from the input documents, and to detect one or moresentences from the input documents. For example, tokenizer andsentencizer 310 may break an input document into words (or words, names,and/or phrases) that are represented by tokens. Additionally, tokenizerand sentencizer 310 may detect sentence breaks in the documents using adependency parser.

Tokens generated by tokenizer and sentencizer 310 may be provided toentity detector 312. Entity detector 312 is configured to receive thetokens and entity list 314 and to determine locations of the entitieswithin the tokens. For example, the entities may be a list of companiesor corporations, and entity detector 312 may determine locations (withina list of tokens, or in a sentence) of one or more entities. As anexample, entity detector 312 may detect a token that corresponds toVerizon and may identify the location of the token. Locations of theentities may also be referred to as entity hit locations. The entity hitlocations may be provided from entity detector 312 to keyword-entitypairing 322.

Tokens generated by tokenizer and sentencizer 310 may also be providedto keyword location detector 316. Keyword location detector 316 may alsoreceive one or more taxonomies and may identify keyword hit locationsand keyword hit text (e.g., which keywords were identified) from thelist of tokens. For example, a keyword in the taxonomy may be “lawsuit”,and keyword location detector 316 may output a location (e.g., in thelist of tokens, or in a sentence) and the text “lawsuit”. The keywordhit locations may be provided to keyword-entity pairing 322, and thekeyword hit text may be pooled together as keyword sentences 318.

The one or more taxonomies provided to keyword location detector 316 mayalso be provided to keyword-based selector 320. Keyword-based selector320 may be configured to select each keyword for determining whetherthere is a keyword-entity match in one of a plurality of input datasources, as further described herein.

Keyword-entity pairing 322 is configured to receive entity hit locationfrom entity detector 312, sentence breaks from tokenizer and sentencizer310, and keyword hit locations from keyword location detector 316.Keyword-entity pairing 322 is configured to determine, for each keyword,the nearest entity to the keyword (and whether it is in the samesentence or not). Keyword-entity pairing 322 may be configured todetermine bi-directional pairings, e.g., the entity that is closest tothe keyword, whether the entity is before or after the keyword. Forexample, if the keyword “lawsuit” is detected, keyword-entity pairing322 may pair the keyword with a first entity that is four tokens beforethe keyword instead of a second entity that is 20 tokens after thekeyword, even if the first entity is in a different sentence. In someimplementations, keyword-entity pairing 322 is configured to operate inparallel such that multiple keywords may be paired with entitiesconcurrently. After pairing keywords and entities, the pairings areprovided to filter 324.

Filter 324 is configured to filter out keyword-entity pairs that aremore than a threshold number of tokens apart. For example, filter 324may compare the distance between the keyword and the entity of akeyword-entity pair to a threshold, and if the distance is greater thanor equal to the threshold, the keyword-entity pair is discarded. In someimplementations, the threshold may be 100 tokens. Additionally, filter324 may assign a score to each filtered keyword-entity pair based on thedistance. For example, a first keyword-entity pair with a lower distancemay have a higher score than a second keyword-entity pair with a higherdistance. The keyword-entity pairs may be mapped to the sentences thatcontain them, and the sentences may be assigned to pools based on thenumber of sentences. For example, if the keyword and the entity are bothin the same sentence, the sentence is assigned to same sentence matchpool 326. Alternatively, if the entity is in a different sentence thanthe keyword, the sentences are assigned to cross-sentence match pool328.

Keyword sentences 318, same sentence match pool 318, and cross-sentencematch pool 328 are used as inputs, based on a keyword selected bykeyword-based selector 320, to output results for comparison. Thecomparison may be between sentence(s) from same sentence match pool 326or cross sentence match pool 328 and a random baseline sentence selectedfrom keyword sentences 318. Comparator 330 may perform a pairwisecomparison between the selected sentence(s) and the baseline sentencebased on scores of the sentences, user input, or a combination thereof.For example, the user input may indicate a rating of each sentence by auser, such as a subject matter expert. The comparisons result inrankings, which are stored as method rankings 332.

To test the performance of system 300 (and/or system 100 of FIG. 1), anexperiment comparing the performances of systems using single-sentencerisk detection and systems only using multi-sentence risks was designed.This experiment measured the performance of purely multi-sentence hitsagainst purely single-sentence hits. In addition, a baseline system wastested that detected only risk terms without searching for correspondingentities. Taken together, the three hypotheses tested are as follows:

H ₁ :p _(multi) >p _(base) ,H _(ø) :p _(multi) =p _(base)

H ₂ :p _(single) >p _(base) ,H _(ø) =p _(single) =p _(base)

H ₃ :p _(multi) >p _(single) ,H _(ø) :p _(multi) =p _(single)

H₁ and H₂ test whether each method of detecting risk co-occurrence withan entity performs better than random chance at selecting a risk term inthe document and assessing it to have been associated with the entity.H₃ tests whether the distance-based measure corresponding to the systemoutperforms a sentential approach.

A virtualized Ubuntu 14.04 machine with 8 vC-PUs—running on 2.30 GHzIntel Xeon E5-2670 processors and 64 GB RAM was chosen to support thefirst experiment.

The names of the top Fortune 100 companies from 2017 were fed as inputinto a series of news requests from Thomson Reuters' CLEAR System toSystem platform for the most recent 1000 articles mentioning eachcompany. Ignoring low coverage and bodiless news articles, 99,424individual documents were returned. Each article was then fed into thesystem and risk detections were found with a distance cutoff of 100tokens. For each identified risk, whether single or multi-sentence, thesystem also selected a baseline sentence at random from thecorresponding document for pairwise comparison.

TABLE 1 Sample risk terms from the seed and expanded sets. EnrichedKeyword Risk Category Keyword Seed Taxonomy Taxonomy Cybersecurity n =26 n = 123 cybercrime, hack, DDOS, 4front security, cyber antivirus,data breach, deterrence, cloakware, ransomware, penetration,unauthenticated, . . . . . . Terrorism n = 37 n = 147 terrorism,bio-terrorism, anti-terrorism, bomb maker, extremist, car bombing,explosives, hezbollah, hijack, guerrilla, . . . jihadi, nationalist, . .. Legal n = 38 n = 162 litigation, indictment, appropriation,concealment, allegation, failure to counter suit, debtor, comply,sanctions expropriation, issuer, . . . violations, . . .

A dependency parse was the largest bottleneck with an expected totalruntime for the near 100,000 documents at approximately 7 calendar-daysof computation. In the interest of runtime, only the first 21,000documents read in order of machine-generated news article ID wereanalyzed. Once all selected documents were processed, single andmulti-sentence spans relating to the same risk category were paired, butpotentially different entities and documents, for pairwise evaluations.

As summarized in Table 1, starting with manually-created seed terms ineach category of risk, encodings were learned from a concatenation ofthe news article text. Selecting the top ten most similar terms for eachin-vocabulary seed term resulted in an expanded taxonomy with a 326.31%increase on average across the three categories. This term expansion notonly introduced new vocabulary to the taxonomy, but also variants andcommon misspellings of keywords, which are important in catching riskterms “in the wild”. Some cleanup of the term expansion was required tofilter out punctuation and tokenization variants.

Analysts were asked to give their preference for “System A” or “SystemB” or “Neither” when presented with randomized pairs of output.Percentage preferences for the overarching system and each of sixpairings was tested for significance with Pearson's, using raw counts.

4514 judgments were collected from eight subject matter experts tocompute system preferences associated with single, multi- and baselinesentence extractions. Roughly 28% of all evaluated extractions(1266/4514) received a preference judgment (32% from the seed set(698/2198) and 24% from the expansion set (568/2316)) where 72% receiveda “Neither” rating.

TABLE 2 Pearson's x² and p values (d.f = 1) System x² Overall  8.530single v. multi (seed v. expand) p = 0.003 CEREALsbs 28.088 single v.baseline (seed) p = 1.159e−07 CEREALmbs 25.358 multi v. baseline (seed)p = 4.762e−07 CEREALsms 37.763 single v. multi (seed) p = 7.99e-10CEREALsbe  6.858 single v. baseline (expand) p = 0.008 CEREALmbe 25.705multi v. baseline (expand) p = 3.978e−07 CEREALsme  6.316 single v.multi (expand) p = 0.011

As summarized in Table 2 (where “CEREAL” corresponds to the system) andby graph 400 of FIG. 4, all single and multi-sentence extractions acrossthe seed and expansion sets outperform the baseline by statisticallysignificant margins. For the seed set, the single sentence extractionsoutperform the multi-sentence extractions by a statistically significantmargin as well (p≤0.01). However, for the expansion set, themulti-sentence extractions gain significant ground (26% to 38% increasein preference).

1283 (28%) of evaluations were doubly annotated for calculation ofCohen's Kappa. Average κ for the seed set was 0.284 and 0.144 for theexpansion set (which suffers from low sample size). This is uniformlylow across all categories, but not an unusual result given the task andthe range of analyst expertise.

The system's distance metric provides benefit well above the baseline,so H₁ and H₂ can be accepted. However, while the high recall is notsurprising, i.e., there is no classification engine and all keywords inthe taxonomy are assumed to capture risk in some context, 72% ofunwanted results need to be effectively managed to not nullify anyanalytical benefits from the preferred results. Adjusting token distancethresholds is one potential way forward that requires furtheranalysis—although this may be subject to document or genre effects. Thecurrent distribution of distances under the 100 threshold are fairlyuniform.

H₃, which tests the assumption that multi-sentence returns will havegreater analytical utility, is not accepted. However, the data indicatesthat as the taxonomy expands, the preference for the multi-sentencesincreases by 46% over the single sentence seed set extractions. In theevaluation set, there were 18 keywords from the expandedtaxonomy—action, assertion, attack, authentication, claim, complaint,compliance, conduct, foreclosure, harassment, identity, military,professionalism, require, requirement, security, separation, and suit.The expanded keywords exhibit a greater range of specificity relative toa risk category than the seed terms. The more specific a risk term is,the less likely it is to appear in ambiguous contexts—arguably why theexpert analyst constructs the seed list as they do. Absent additionalinformation, the more general the risk term, the higher likelihood ofambiguity. It seems here that additional study is potentially requiredto understand the nature of the risk of the expanded keywords because oftheir general nature. Based on the proliferation of more general termsin the expanded set, perhaps cull high recall, the thresholding shouldconsider the degree of semantic granularity as well.

Thus, the system described herein includes a scalable system for findingentity-event pairings which, as compared to other systems, betterincorporates human insights in the initial system configuration toobviate the need for a risk classification engine. Further, withoutrelying on multiple complex NLP sub-systems to make connections betweenentities and event keywords, the system may derive maximum benefit fromsurface information.

While applied here to the risk mining space, in other implementations,the system could be used for any data, entities and taxonomies tosupport generalized event monitoring. Additionally, the system mayaddress high system recall relative to maintaining flexibility foranalyst users and dynamic definition of the risk problem space—this mayinclude summarization of results for better presentation, alternativesource data at the direction of the analyst for given risk categories,and token distance thresholding.

End Example

Referring to FIG. 5 is a flow diagram of a method 500 of identifying anevent in data. In some implementations, the method 400 may be performedby system 100 of FIG. 1 (e.g.,), one or more components to executeoperations of FIG. 2, or system 300 of FIG. 3.

Method 500 includes receiving data at a receiver from a data source, thedata comprising one or more documents each comprising text, at block502. For example, data ingestor 120 may receive documents (e.g., thedata) from data sources 170.

Method 500 includes performing natural language processing on thereceived data to generate processed data, the processed data indicatingone or more sentences, at block 504. For example, data ingestor 120 mayperform natural language processing on the received documents togenerate processed data that indicates one or more sentences.

Method 500 also includes generating, based on the data and a firstkeyword set, a second keyword set having a greater number of keywordsthan the first keyword set, at block 506. For example, taxonomy expander124 may generate a second keyword set having a greater number ofkeywords than a first keyword set (e.g., a seed set).

Method 500 further includes apply each keyword set of the first keywordset and the second keyword set to the processed data, at block 508.Applying each keyword set may include one or more blocks, such as blocks510, 512, 514, and 516, as described further herein.

Method 500 further includes detecting one or more keywords and one ormore entities included in the processed data based on the keyword setand an entity set, at block 510. For example, term detector 121 maydetect one or more keywords and one or more entities included in theprocessed data based on the keyword set and an entity set.

Method 500 further includes determining one or more matched pairs basedon the detected one or more keywords and the detected one or moreentities, at block 512. For example, output generator 122 may determineone or more matched pairs based on the detected one or more keywords andthe detected one or more entities.

Method 500 further includes extracting a sentence from a document basedon the one or more sentences indicated by the processed data, where thesentence corresponds to at least one matched pair of the one or morematched pairs and comprises a single sentence or multiple sentences, atblock 514. For example, output generator 122 may extract a sentence froma document based on the one or more sentences indicated by the processeddata, such that the sentence corresponds to at least one matched pair.The sentence may be a single sentence or multiple sentences (e.g., if amatched pair crosses two or more sentences).

Method 500 further includes outputting the extracted sentence, at block516. For example, output generator 122 may output the extractedsentence, for example, for display to a user. Additionally, oralternatively, the extracted sentence may be stored at a memory.

In some implementations, performing the natural language processingincludes performing, on the data, tokenization, lemmatization,sentencization, or a combination thereof. For example, data ingestor 102may perform tokenization, lemmatization, sentencization, or acombination thereof, on data from data sources 170. Additionally, oralternatively, method 500 may further comprise, for each matched pairbased on the first keyword set and each matched pair based on the secondkeyword set, determining whether the keyword and the entity of thematched pair are included in the same sentence. Additionally, extractingthe sentence from the document includes extracting a single sentencebased on a determination that the keyword and the entity are included inthe same sentence and extracting multiple sentences based on adetermination that the keyword and the entity are not included in thesame sentence. For example, if the keyword and the entity of a matchedpair are in the same sentence, output generator 122 may extract a singlesentence. Alternatively, if the keyword and the entity of a matched pairare in different sentences, output generator 122 may extract multiplesentences.

In some implementations, method 500 further includes initiating a pipe() operation on the data to perform the natural language processing.Performing the natural language processing further includes using adependency based sentencizer to generate the processed data indicatingone or more sentences, converting the data to a lemmatized format, usinga tokenizer to generate one or more tokens, using a part-of-speechtagger to generate part-of-speech data, using a named entity recognizerto identify one or more entities, or a combination thereof. In some suchimplementations, the processed data is in a format that is compatiblewith Python. For example, data ingestor 120 may include a dependencybased sentencizer, a lemmatizer, a tokenizer, a part-of-speech tagger, anamed entity recognizer, or a combination thereof.

In some implementations, generating the second keyword set includesgenerating one or more semantic vectors. Generating the second keywordset also includes, for each keyword of the first keyword set:determining a semantic vector having a highest similarity score to thekeyword and identifying one or more terms of the determined semanticvector as a candidate term. Generating the second keyword set alsoincludes selecting at least one candidate term to be added to the firstkeyword set to generate the second keyword set. For example, taxonomyexpander 124 may generate semantic vectors and identify terms ofsemantic vectors as candidate terms based on similarity scores. In somesuch implementations, generating the one or more semantic vectorsincludes, for each document of one or more documents corresponding tothe data, generating a corresponding semantic vector based on a skipgrammodel that utilizes words and subwords from the document. For example, askipgram generator, such as Fasttext, may be used to generate thesemantic vectors. Generating the second keyword set further includes,for each keyword of the first keyword set, comparing a similarity scoreof the determined semantic vector having a highest similarity score to athreshold. The semantic vector is used to identify the candidate termbased on a determination that the similarity score of the determinedsemantic vector is greater than or equal to the threshold.

In some implementations, the extracted sentence is output to anelectronic device associated with an analyst, and method 500 furtherincludes receiving an input from the analyst responsive to the extractedsentence, storing an indication of the input, and sending a notificationcorresponding to the extracted sentence, the input, or both. Thenotification includes a link to a data source, a text extraction from adocument, the matched pair corresponding to the extracted sentence, theinput, an identifier of the analyst, or a combination thereof.

In some implementations, method 500 further includes receiving aselection of a first event category of multiple event categories andretrieving the first keyword set based on the selection of the firstevent category. For example, different keyword sets may correspond todifferent event categories. To illustrate, one keyword set maycorrespond to “terrorism” and another keyword set may correspond to“legal.” In some such implementations, the multiple event categoriesinclude cybersecurity, terrorism, legal/non-compliance, or a combinationthereof. In some such implementations, method 500 further includesreceiving a selection of a second event category of the multiple eventcategories, retrieving a third keyword set based on the selection of thesecond event category, generating, based on the third keyword set, afourth keyword set having a greater number of keywords than the thirdkeyword set, and, for each keyword set of the third keyword set and eachkeyword in the fourth keyword set: detecting one or more keywords andone or more entities included in the processed data based on the keywordset and the entity set and determining one or more matched pairs basedon the detected one or more keywords and the detected one or moreentities. For example, different taxonomies corresponding to differententity lists may be generated and used to determine keyword-entitypairs.

In some implementations, the sentence includes the multiple sentences,and the multiple sentences include a sentence that includes the at leastone matched pair, a sentence that includes the keyword of the at leastone matched pair, a sentence preceding the sentence that includes thekeyword of the at least one matched pair, a sentence following thesentence with the keyword the at least one matched pair, a sentence thatincludes the entity of the at least one matched pair, a sentencepreceding the sentence that includes the entity of the at least onematched pair, a sentence following the sentence with the entity of theat least one matched pair, or a combination thereof. Additionally, oralternatively, the data source includes a streaming data source, newsdata, a database, or a combination thereof, and the entity set indicatesan individual, a company, a government, an organization, or acombination thereof.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the disclosure herein may be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present disclosure. Skilled artisans will also readilyrecognize that the order or combination of components, methods, orinteractions that are described herein are merely examples and that thecomponents, methods, or interactions of the various aspects of thepresent disclosure may be combined or performed in ways other than thoseillustrated and described herein.

Functional blocks and modules in FIGS. 1-5 may comprise processors,electronics devices, hardware devices, electronics components, logicalcircuits, memories, software codes, firmware codes, etc., or anycombination thereof. Consistent with the foregoing, various illustrativelogical blocks, modules, and circuits described in connection with thedisclosure herein may be implemented or performed with a general-purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

The steps of a method or algorithm described in connection with thedisclosure herein may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such that theprocessor can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor. The processor and the storage medium may reside in anASIC. The ASIC may reside in a user terminal, base station, a sensor, orany other communication device. In the alternative, the processor andthe storage medium may reside as discrete components in a user terminal.

In one or more exemplary designs, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on acomputer-readable medium. Computer-readable media includes both computerstorage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another.Computer-readable storage media may be any available media that can beaccessed by a general purpose or special purpose computer. By way ofexample, and not limitation, such computer-readable media can compriseRAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium that canbe used to carry or store desired program code means in the form ofinstructions or data structures and that can be accessed by ageneral-purpose or special-purpose computer, or a general-purpose orspecial-purpose processor. Also, a connection may be properly termed acomputer-readable medium. For example, if the software is transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, or digital subscriber line (DSL), thenthe coaxial cable, fiber optic cable, twisted pair, or DSL, are includedin the definition of medium. Disk and disc, as used herein, includescompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods, and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

1. A method for identifying an event in data, the method comprising:receiving data at a receiver from a data source, the data comprising oneor more documents each comprising text; performing natural languageprocessing on the received data to generate processed data, theprocessed data indicating one or more sentences; generating, based onthe data and a first keyword set, a second keyword set having a greaternumber of keywords than the first keyword set; and for each keyword setof the first keyword set and each keyword in the second keyword set:detecting one or more keywords and one or more entities included in theprocessed data based on the keyword set and an entity set; determiningone or more matched pairs based on the detected one or more keywords andthe detected one or more entities; extracting a sentence from a documentbased on the one or more sentences indicated by the processed data,where the sentence corresponds to at least one matched pair of the oneor more matched pairs and comprises a single sentence or multiplesentences; and outputting the extracted sentence.
 2. The method of claim1, wherein performing the natural language processing comprisesperforming, on the data, tokenization, lemmatization, sentencization, ora combination thereof.
 3. The method of claim 1, further comprising, foreach matched pair based on the first keyword set and each matched pairbased on the second keyword set: determining whether the keyword and theentity of the matched pair are included in the same sentence; andwherein extracting the sentence from the document comprises: extractinga single sentence based on a determination that the keyword and theentity are included in the same sentence; and extracting multiplesentences based on a determination that the keyword and the entity arenot included in the same sentence.
 4. The method of claim 1, furthercomprising: initiating a pipe( ) operation on the data to perform thenatural language processing; and wherein performing natural languageprocessing further comprising: using a dependency based sentencizer togenerate the processed data indicating one or more sentences; convertingthe data to a lemmatized format; using a tokenizer to generate one ormore tokens; using a part-of-speech tagger to generate part-of-speechdata; using a named entity recognizer to identify one or more entities;or a combination thereof; and wherein the processed data is in a formatthat is compatible with Python.
 5. The method of claim 1, whereingenerating the second keyword set comprises: generating one or moresemantic vectors; for each keyword of the first keyword set: determininga semantic vector having a highest similarity score to the keyword;identifying one or more terms of the determined semantic vector as acandidate term; and selecting at least one candidate term to be added tothe first keyword set to generate the second keyword set.
 6. The methodof claim 5, wherein: generating the one or more semantic vectorscomprises, for each document of one or more documents corresponding tothe data, generating a corresponding semantic vector based on a skipgrammodel that utilizes words and subwords from the document; and generatingthe second keyword set further comprises, for each keyword of the firstkeyword set: comparing a similarity score of the determined semanticvector having a highest similarity score to a threshold; and wherein thesemantic vector is used to identify the candidate term based on adetermination that the similarity score of the determined semanticvector is greater than or equal to the threshold.
 7. The method of claim1, wherein the extracted sentence is output to an electronic deviceassociated with an analyst, and further comprising: receiving an inputfrom the analyst responsive to the extracted sentence; and storing anindication of the input; and sending a notification corresponding to theextracted sentence, the input, or both; and wherein the notificationincludes a link to a data source, a text extraction from a document, thematched pair corresponding to the extracted sentence, the input, anidentifier of the analyst, or a combination thereof.
 8. A systemcomprising: a data ingestor configured to: receive data at a receiverfrom a data source, the data comprising one or more documents eachcomprising text; and perform natural language processing on the receiveddata to generate processed data, the processed data indicating one ormore sentences; a taxonomy expander configured to: generate, based onthe data and a first keyword set, a second keyword set having a greaternumber of keywords than the first keyword set; a term detectorconfigured to: detect, for each keyword set of the first keyword set andeach keyword in the second keyword set, one or more keywords and one ormore entities included in the processed data based on the keyword setand an entity set; and an output generator configured to, for eachkeyword set of the first keyword set and the second keyword set:determine one or more matched pairs based on the detected one or morekeywords and the detected one or more entities; extract a sentence froma document based on the one or more sentences indicated by the processeddata, where the sentence corresponds to at least one matched pair of theone or more matched pairs and comprises a single sentence or multiplesentences; and output the extracted sentence.
 9. The system of claim 8,further comprising: a database coupled to the data ingestor, thetaxonomy expander, the term detector, the output generator, or acombination thereof.
 10. The system of claim 9, wherein the database isconfigured to store the first keyword set, the second keyword set, theentity set, the processed data, one or more thresholds, one or moreextracted sentences, a plurality of matched pairs, or a combinationthereof.
 11. The system of claim 8, further comprising: a processor; anda memory storing instructions executable by the processor to cause theprocessor to perform one or more operations of the data ingestor, thetaxonomy expander, the term detector, the output generator, or acombination thereof.
 12. The system of claim 8, further comprising: aninterface configured to enable communication with the data source, anelectronic device, or a combination thereof.
 13. The system of claim 8,further comprising: a filter configured to: for each of the one or morematched pairs based on the first keyword set and each of the one or morematched pairs based on the second keyword set: determine a distancebetween the keyword and the entity of the matched pair; perform acomparison between the determined distance and a threshold; anddetermine to retain the matched pair or discard the matched pair basedon whether or not the comparison indicate the determined distance isgreater than or equal to the threshold.
 14. The system of claim 8,wherein, to determine the one or more matched pairs, the outputgenerator is further configured to: for each keyword of the detected oneor more keywords, identify a corresponding entity of the detected one ormore entities that is positioned closest to the corresponding keyword todetermine a matched pair for the keyword.
 15. A computer-based toolincluding non-transitory computer readable media having stored thereoncomputer code which, when executed by a processor, causes a computingdevice to perform operations comprising: receiving data at a receiverfrom a data source, the data comprising one or more documents eachcomprising text; performing natural language processing on the receiveddata to generate processed data, the processed data indicating one ormore sentences; generating, based on the data and a first keyword set, asecond keyword set having a greater number of keywords than the firstkeyword set; for each keyword of the first keyword set and each keywordin the second keyword set: detecting one or more keywords and one ormore entities included in the processed data based on the keyword setand an entity set; determining one or more matched pairs based on thedetected one or more keywords and the detected one or more entities;extracting a sentence from a document based on the one or more sentencesindicated by the processed data, where the sentence corresponds to atleast one matched pair of the one or more matched pairs and comprises asingle sentence or multiple sentences; and outputting the extractedsentence.
 16. The computer-based tool of claim 15, wherein theoperations further comprise: receiving a selection of a first eventcategory of multiple event categories; and retrieving the first keywordset based on the selection of the first event category.
 17. Thecomputer-based tool of claim 16, wherein the multiple event categoriescomprise cybersecurity, terrorism, legal/non-compliance, or acombination thereof.
 18. The computer-based tool of claim 16, whereinthe operations further comprise: receiving a selection of a second eventcategory of the multiple event categories; retrieving a third keywordset based on the selection of the second event category; generating,based on the third keyword set, a fourth keyword set having a greaternumber of keywords than the third keyword set; for each keyword set ofthe third keyword set and each keyword in the fourth keyword set:detecting one or more keywords and one or more entities included in theprocessed data based on the keyword set and the entity set; anddetermining one or more matched pairs based on the detected one or morekeywords and the detected one or more entities.
 19. The computer-basedtool of claim 15, wherein: the sentence comprises the multiplesentences; and the multiple sentences comprise a sentence that includesthe at least one matched pair, a sentence that includes the keyword ofthe at least one matched pair, a sentence preceding the sentence thatincludes the keyword of the at least one matched pair, a sentencefollowing the sentence with the keyword the at least one matched pair, asentence that includes the entity of the at least one matched pair, asentence preceding the sentence that includes the entity of the at leastone matched pair, a sentence following the sentence with the entity ofthe at least one matched pair, or a combination thereof.
 20. Thecomputer-based tool of claim 15, wherein: the data source comprises astreaming data source, news data, a database, or a combination thereof;and the entity set indicates an individual, a company, a government, anorganization, or a combination thereof.